Source of Acquisition 
NASA Johnson Space Center 


Making Robonaut An Intelligent Assistant for Humans 

Abstract 

This presentation is an overview of the Robonaut project. Robonaut is a 
humanoid robot designed by the Automation, Robotics, and Simulation Division 
at NASA's Johnson Space Center. The Robonaut project seeks to develop and 
demonstrate a robotic system that can function as an EVA astronaut equivalent. 
Robonaut jumps generations ahead by eliminating the robotic scars (e.g., special 
robotic grapples and targets) and specialized robotic tools of traditional on-orbit 
robotics. However, it still keeps the human operator in the control loop through 
its telepresence control system. Robonaut is being designed for "EVA" tasks, 
i.e., those that were not specifically designed for robots. 

Our challenge is to build machines that can help humans work and explore in 
space. Working side by side with humans, or going where the risks are too great 
for people, machines like Robonaut will expand our ability for construction and 
discovery. Central to that effort is a capability we call dexterous manipulation, 
embodied by an ability to use ones hand to do work, and our challenge is to build 
machines with dexterity that exceeds that of a suited astronaut. 
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ROBONAUT Progress 



Thrusts during FY01 

- Autonomy 

• Added vision 

• Added voice 

• Added sequence logic 

- Collaboration 

• Created RoboSim 

• Created RoboAPI 

- 3 Human Factors Studies 

- 2 Human/Robot Team Studies 

Recent Work in FY02 

- Multi tool identification 

- Human Tracking 

- Tool Exchange 

- New Human/Robot Team Study 


ROBONAUT Fall 2001 


ROBONAUT 
Fall 1998 


ROBONAUT 
Fall 1999 


ROBONAUT 
Fall 2000 



NASA Teams Humans and Robots 






These teams are in orbit 
today. 

NASA has approaching 20 
years of EVA work with 
robotic systems. 

Mission developers are 
now calling for 
expansions of these teams 
with more robots, new 
forms of robots, and new 
capabilities for assisting 
humans in space. 



Photo of CanadArm on STS 103 and Lunar Rover 


Taxonomy of a Team 


Established Dimensions of a Team 

• Spatial relations (remote operation, shoulder-to-shoulder) 

• Ratios of agents (ratios of subordinates) 

• Relationships of rank (command hierarchies) 

Now being Investigated at JSC 

• Heterogeneous agents (sentries, scouts, transport, shock, artillery) 

• Forms of interactions (data, hand offs, sustained physical contact) 


Heterogeneous Teams, Today 


• Current EVA Teams 

- 2 EVA Humans 

- 1 RMS 

- 1 IV A RMS Operator 

- 1 IVA “watcher” 

- 10-30 Ground Personnel 

• Team limits and strengths 

- >150 EVA in human history 

- Hugely successful 

- No losses to date 

- Consumes full flight crew 

- 1-3 Hours of prep time 

- 8 Hour EVA time limit 

- 1-3 Hours of cleanup 




Heterogeneous Teams, Tomorrow 








New EVA practices 

- 2 EVA Humans 

- 1 RMS 

- 1 Robonaut 

- 1 Aercam 

- 1 IVA RMS Operator 

- 1 IVA Robonaut operator 

- 1 IVA Aercam Operator 

- 10-30 Ground Support 

Team limits and strengths 

- Requires more than available crew 

- Allows Humans to “split up” 





Forms of Human/Robot Interaction 






Information Connections 

- Levels of Intervention 

• Autonomous 

• Supervised 

• Teleoperated 

- Forms of Communication 

• Voice 

• Natural gestures 

• Input devices 

• Sensor Feedback 

• State & health 

Physical Connections 

- Presence (in situ) 

- Intermittent contact (hand off) 

- Coordinated contact (work) 





Agent Interaction: Teleoperation 



Command 

Data 


Agents are connected by 
information alone 



X 


Video & Sensor 
Feedback 




Robots and Remote Humans 


Completed 4 human 
interface experiments 

- MIT: Jen Rochlis working 
on visual displays 

- NASDA: Sachiko 
Wakabayashi working on 
constrained motion 

- Univ. of Houston: Lore 
Williams working on force 
feedback 

- RIT: Julie Adams working 
on qualitative modeling of 
robot health and state. 

Now upgrading interface 



Photo of Jen (June 01) 
during her experiment 


Photo of Lore (July 0 1 ) 
during her experiment 


Photo of Sachiko (Nov 01) 
during her experiment 



Agent Interaction: Autonomous 
Assistant for Human 


Agents interact physically 
and with information flow 


Bi-Directional 
4 ► Data 



4 ► Physical 

Interaction 



Autonomy: Handing Tool to Human 






Autonomy: Human Tracking 


Autonomy: Tool Tracking 




Agent Interaction: Teleoperated 
Assistant for Human 



Information 


Information 


3 Heterogeneous agents with 
unique, multi-modal interactions 


Physical Connection 


Supervised Assistant: Tether Handling 




I 



Supervised Assistant: Soldering 





Supervised Assistant: Multimeter 



Supervised Assistant: Construction 






Agent Interaction: Supervised 
Assistant for Human 



Information 


Information 


3 Heterogeneous agents with 
unique, multi-modal interactions 
and intervention levels 


Information & Physical Connection 


Robonautonomy 


Objective: 

“ The automation of dexterous manipulation for 
ROBONAUT class humanoids, allowing 
supervisory levels of command from adjacent 
human team mates.” 

Motivation: 

-Characteristic distances in space limit information 
flow needed for interaction. 

-Enable more work with a small, in situ workforce 


Distributed Approach for R&D 


Avoid R&D constraints 

- Packaging 

- OS incompatibilities 

- Power, Comm, Hardening 

Enable collaboration 

- Open to HW/SW spectrum 

- Provide I/O pathways 

Functional Lobes 

- Vision 

- Grasping 

- Data storage 

- Motion coordination 

- Sensor I/O 


^eaf n,n 9 & Reasoning 

I Limb Coordination I 



Hippocampus 


Cerebellum 


Visual Cortex 

(Memory) 


(Motion) 


(Active Vision) 













Robonautonomy: Vision 


• Human tracking 

- Human identification 

- Limb tracking 

• Object Tracking 

- Tool identification 

- Tool pose estimation 

- Tool occlusion 

• Integration 

- Integrated with SES 

- Integrated with motion 
control 

- Integrated with eyes 




Recent FY02 Advances 


Find a tool within a field of 
many tools 


Search for and find a tool, 
partially occluded by a hand 


Give and take a tool from a 
human commander 


Collaboration 







RoboSIM & RoboAPI 

DARPA Partners 

- Vanderbilt (SES integrated) 

- UMass (Grasping in progress) 

- Invitations offered 

• USC, MIT, over 20 others 

NASA has avenues for teaming 

- Summer faculty programs 

- Graduate student fellowships 

- Coop and Intern programs 




Screen shot of ROBONAUT Simulation running on PC 


The ROBONA UT Team 




